Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Osquerybeat: Add missing data_stream to events in order to support logstash configuration better #32543

Merged
merged 1 commit into from Aug 1, 2022
Merged

Osquerybeat: Add missing data_stream to events in order to support logstash configuration better #32543

merged 1 commit into from Aug 1, 2022

Conversation

aleksmaus
Copy link
Member

What does this PR do?

Adds the add_data_stream processors that appends the data_stream fields to the events. This enabled better compatibility with the logstash configuration allowing it to figure out the destination datastream for the results of the osquery.

Why is it important?

Better support osquery with logstash as the output.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas

How to test this PR locally

Here is an example logstash pipeline configuration that was not working before this change:

input {
  elastic_agent {
    port => 5044
    ssl => true
    ssl_certificate_authorities => ["certs/ca/ca.crt"]
    ssl_certificate => "certs/logstash.crt"
    ssl_key =>  "certs/logstash.pkcs8.key"
    ssl_verify_mode => "force_peer"
  }
}

filter {
    mutate {
        rename => ["_host", "host" ]
    }
}

output {
  elasticsearch {
    hosts => "https://<redacted>.us-west2.gcp.elastic-cloud.com:443"
    data_stream => true
    ssl => true
    user => elastic
    password => <redacted>
  }
}

After the change the event has additional datastream attributes

 "data_stream" => {
    "namespace" => "default",
    "type" => "logs",
    "dataset" => "osquery_manager.result"
},

Screenshots

Verified the result are now reaching the osquery_manager datastream:

Screen Shot 2022-07-28 at 6 08 15 PM

@aleksmaus aleksmaus requested a review from a team as a code owner July 28, 2022 22:36
@aleksmaus aleksmaus requested review from faec and leehinman and removed request for a team July 28, 2022 22:36
@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Jul 28, 2022
@cmacknz cmacknz added backport-v8.4.0 Automated backport with mergify and removed backport-v8.4.0 Automated backport with mergify labels Jul 28, 2022
@elasticmachine
Copy link
Collaborator

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2022-07-28T22:36:33.574+0000

  • Duration: 48 min 58 sec

Test stats 🧪

Test Results
Failed 0
Passed 1620
Skipped 0
Total 1620

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

  • /package : Generate the packages and run the E2E tests.

  • /beats-tester : Run the installation tests with beats-tester.

  • run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

@aleksmaus aleksmaus added the backport-v8.4.0 Automated backport with mergify label Jul 29, 2022
patrykkopycinski
patrykkopycinski approved these changes Aug 1, 2022
View reviewed changes
Copy link

@patrykkopycinski patrykkopycinski left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❤️

@aleksmaus aleksmaus merged commit b26092f into elastic:main Aug 1, 2022
mergify bot pushed a commit that referenced this pull request Aug 1, 2022
@aleksmaus @mergify
Osquerybeat: Add missing data_stream to events in order to support lo…
f283fee 
…gstash default configuration better (#32543)

(cherry picked from commit b26092f)
@mergify mergify bot mentioned this pull request Aug 1, 2022
Merged
aleksmaus added a commit that referenced this pull request Aug 1, 2022
@mergify @aleksmaus
Osquerybeat: Add missing data_stream to events in order to support lo…
eb85bc9 
…gstash default configuration better (#32543) (#32562)

(cherry picked from commit b26092f)

Co-authored-by: Aleksandr Maus <aleksandr.maus@elastic.co>
chrisberkhout pushed a commit that referenced this pull request Jun 1, 2023
@aleksmaus @chrisberkhout
Osquerybeat: Add missing data_stream to events in order to support lo…
c2ac022 
…gstash default configuration better (#32543)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@patrykkopycinski patrykkopycinski patrykkopycinski approved these changes

@melissaburpo melissaburpo Awaiting requested review from melissaburpo

@james-elastic james-elastic Awaiting requested review from james-elastic

@faec faec Awaiting requested review from faec faec was automatically assigned from elastic/elastic-agent-data-plane

@leehinman leehinman Awaiting requested review from leehinman leehinman was automatically assigned from elastic/elastic-agent-data-plane

Assignees

@aleksmaus aleksmaus

Labels
8.4-candidate backport-v8.4.0 Automated backport with mergify enhancement Team:Asset Mgt
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@aleksmaus @elasticmachine @patrykkopycinski @cmacknz